This repository was archived by the owner on Jul 16, 2025. It is now read-only.
feat: implement document loader & transformer for store indexing #343
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
OskarStark
reviewed
Jun 26, 2025
chr-hertel
force-pushed
the
feat-loader-transformer-pipeline
branch
from
7e27551 to
d97f44e
Compare
chr-hertel
changed the title
chr-hertel
force-pushed
the
feat-loader-transformer-pipeline
branch
from
d97f44e to
c76251f
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR implements a document loader and transformer for store indexing while refactoring the Indexer to use batch processing with the new Vectorizer and adding several tests for the document splitting and loading functionalities.
- Refactored Indexer to remove clock-based sleep logic and integrate batch vectorization via the Vectorizer.
- Added comprehensive tests for TextSplitTransformer, ChainTransformer, and TextFileLoader.
- Introduced new loader and transformer interfaces and implementations.
Reviewed Changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/Store/IndexerTest.php | Renamed test methods and removed the dependency on MockClock. |
| tests/Store/Document/Transformer/TextSplitTransformerTest.php | Added tests to validate document splitting behavior with various parameters. |
| tests/Store/Document/Transformer/ChainTransformerTest.php | Added tests to ensure chaining of document transformers works as expected. |
| tests/Store/Document/Loader/TextFileLoaderTest.php | Added tests for handling valid/invalid file sources and metadata inclusion. |
| src/Store/Indexer.php | Refactored to incorporate batch vectorization using Vectorizer but introduced a potential issue with processing remaining chunks. |
| src/Store/Document/Vectorizer.php | Introduced a separate component to handle document vectorization. |
| src/Store/Document/Transformer/* | New transformer components (TextSplit, ChunkDelay, ChainTransformer) added. |
| src/Store/Document/Loader/* | New TextFileLoader and LoaderInterface implementations added. |
| examples/store/document-splitting.php | Added example demonstrating document splitting usage. |
chr-hertel
force-pushed
the
feat-loader-transformer-pipeline
branch
3 times, most recently
from
8f5c001 to
6e484a2
Compare
OskarStark
approved these changes
Jun 28, 2025
chr-hertel
force-pushed
the
feat-loader-transformer-pipeline
branch
from
6e484a2 to
7e2d5ba
Compare
chr-hertel
added a commit
to symfony/ai
that referenced
this pull request
Jul 4, 2025
…ndexing (chr-hertel) This PR was merged into the main branch. Discussion ---------- feat: implement document loader & transformer for store indexing | Q | A | ------------- | --- | Bug fix? | no | New feature? | yes | Docs? | | Issues | | License | MIT Cherry picking php-llm/llm-chain#343 Commits ------- 83ce86f feat: implement document loader & transformer for store indexing (#343)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.